{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started\n",
    "\n",
    "This notebook gets you started with a brief nDCG evaluation with LensKit for Python.\n",
    "\n",
    "This notebook is also available on [Google Collaboratory](https://colab.research.google.com/drive/1ym040cKkQf85epu80VtIkMXy3LpfYQky?usp=sharing) and [nbviewer](https://nbviewer.jupyter.org/github/lenskit/lkpy/blob/master/doc/GettingStarted.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "We first import the LensKit components we need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lenskit.datasets import ML100K\n",
    "from lenskit import batch, topn, util\n",
    "from lenskit import crossfold as xf\n",
    "from lenskit.algorithms import Recommender, als, item_knn as knn\n",
    "from lenskit import topn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And Pandas is very useful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Data\n",
    "\n",
    "We're going to use the ML-100K data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>item</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "      <td>881250949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "      <td>891717742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "      <td>878887116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "      <td>880606923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "      <td>886397596</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  item  rating  timestamp\n",
       "0   196   242       3  881250949\n",
       "1   186   302       3  891717742\n",
       "2    22   377       1  878887116\n",
       "3   244    51       2  880606923\n",
       "4   166   346       1  886397596"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml100k = ML100K('ml-100k')\n",
    "ratings = ml100k.ratings\n",
    "ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining Algorithms\n",
    "\n",
    "Let's set up two algorithms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "algo_ii = knn.ItemItem(20)\n",
    "algo_als = als.BiasedMF(50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running the Evaluation\n",
    "\n",
    "In LensKit, our evaluation proceeds in 2 steps:\n",
    "\n",
    "1. Generate recommendations\n",
    "2. Measure them\n",
    "\n",
    "If memory is a concern, we can measure while generating, but we will not do that for now.\n",
    "\n",
    "We will first define a function to generate recommendations from one algorithm over a single partition of the data set.  It will take an algorithm, a train set, and a test set, and return the recommendations.\n",
    "\n",
    "**Note:** before fitting the algorithm, we clone it.  Some algorithms misbehave when fit multiple times.\n",
    "\n",
    "**Note 2:** our algorithms do not necessarily implement the `Recommender` interface, so we adapt them. This fills in a default candidate selector.\n",
    "\n",
    "The code function looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def eval(aname, algo, train, test):\n",
    "    fittable = util.clone(algo)\n",
    "    fittable = Recommender.adapt(fittable)\n",
    "    fittable.fit(train)\n",
    "    users = test.user.unique()\n",
    "    # now we run the recommender\n",
    "    recs = batch.recommend(fittable, users, 100)\n",
    "    # add the algorithm name for analyzability\n",
    "    recs['Algorithm'] = aname\n",
    "    return recs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will loop over the data and the algorithms, and generate recommendations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_recs = []\n",
    "test_data = []\n",
    "for train, test in xf.partition_users(ratings[['user', 'item', 'rating']], 5, xf.SampleFrac(0.2)):\n",
    "    test_data.append(test)\n",
    "    all_recs.append(eval('ItemItem', algo_ii, train, test))\n",
    "    all_recs.append(eval('ALS', algo_als, train, test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the results in place, we can concatenate them into a single data frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item</th>\n",
       "      <th>score</th>\n",
       "      <th>user</th>\n",
       "      <th>rank</th>\n",
       "      <th>Algorithm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>285</td>\n",
       "      <td>4.543364</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>ItemItem</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1449</td>\n",
       "      <td>4.532999</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>ItemItem</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1251</td>\n",
       "      <td>4.494639</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>ItemItem</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>114</td>\n",
       "      <td>4.479512</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>ItemItem</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>4.399639</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>ItemItem</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   item     score  user  rank Algorithm\n",
       "0   285  4.543364     5     1  ItemItem\n",
       "1  1449  4.532999     5     2  ItemItem\n",
       "2  1251  4.494639     5     3  ItemItem\n",
       "3   114  4.479512     5     4  ItemItem\n",
       "4   166  4.399639     5     5  ItemItem"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_recs = pd.concat(all_recs, ignore_index=True)\n",
    "all_recs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute our analysis, we also need to concatenate the test data into a single frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data = pd.concat(test_data, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We analyze our recommendation lists with a `RecListAnalysis`.  It takes care of the hard work of making sure that the truth data (our test data) and the recoommendations line up properly.\n",
    "\n",
    "We do assume here that each user only appears once per algorithm. Since our crossfold method partitions users, this is fine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/MICHAELEKSTRAND/anaconda3/envs/lkpy-dev/lib/python3.7/site-packages/pandas/core/indexing.py:1494: PerformanceWarning: indexing past lexsort depth may impact performance.\n",
      "  return self._getitem_tuple(key)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>ndcg</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user</th>\n",
       "      <th>Algorithm</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
       "      <th>ALS</th>\n",
       "      <td>0.265268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ItemItem</th>\n",
       "      <td>0.259708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
       "      <th>ALS</th>\n",
       "      <td>0.148335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ItemItem</th>\n",
       "      <td>0.081890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <th>ALS</th>\n",
       "      <td>0.026615</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ndcg\n",
       "user Algorithm          \n",
       "1    ALS        0.265268\n",
       "     ItemItem   0.259708\n",
       "2    ALS        0.148335\n",
       "     ItemItem   0.081890\n",
       "3    ALS        0.026615"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rla = topn.RecListAnalysis()\n",
    "rla.add_metric(topn.ndcg)\n",
    "results = rla.compute(all_recs, test_data)\n",
    "results.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have nDCG values!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Algorithm\n",
       "ALS         0.139689\n",
       "ItemItem    0.102075\n",
       "Name: ndcg, dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.groupby('Algorithm').ndcg.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f03842f8860>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEtCAYAAADk97CmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFVFJREFUeJzt3X+w5XV93/Hny10hghEM3NjKLu6mrO0sP2L0suo0wQYaXJIpa0dId8k00KHdtkonMzZtVptisjqtdBpJpsGpm0AkqLMQjM122IiJKK2ppbugAitucl0JXNdpVkEiWlgX3v3jfHc8HC/c7929ew7cz/Mxc2e/38/38/2e92Eur/O5n/P9kapCktSGF026AEnS+Bj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYsn3QBo0499dRatWrVpMuQpBeUu++++xtVNTVfv+dd6K9atYrdu3dPugxJekFJ8pd9+jm9I0kNMfQlqSGGviQ1xNCXpIYY+pLUkF6hn2R9kr1JZpJsmWP7eUnuSXIoySVzbH9Zkq8l+e3FKFqSdGTmDf0ky4DrgIuAtcCmJGtHuj0EXAF89FkO8x7gziMvU5K0GPqM9NcBM1W1r6oOAtuBDcMdqurBqroXeHp05ySvA14BfHIR6pUkHYU+oX8a8PDQ+mzXNq8kLwJ+A/g38/TbnGR3kt0HDhzoc2hJ0hHoc0Vu5mjr+zT1twE7q+rhZK7DdAer2gZsA5ienn5BPKl91ZbbJl3CkvLg+35u0iVITegT+rPAyqH1FcD+nsd/I/BTSd4GvBQ4LsnjVfUDXwZLko69PqG/C1iTZDXwNWAjcFmfg1fVLxxeTnIFMG3gS9LkzDunX1WHgKuA24EHgFuqak+SrUkuBkhybpJZ4FLgg0n2HMuiJUlHptddNqtqJ7BzpO3qoeVdDKZ9nusYHwI+tOAKJUmLxityJakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUkF6hn2R9kr1JZpJsmWP7eUnuSXIoySVD7a9J8rkke5Lcm+QfLWbxkqSFmTf0kywDrgMuAtYCm5KsHen2EHAF8NGR9u8Cv1hVZwLrgd9McvLRFi1JOjLLe/RZB8xU1T6AJNuBDcCXDneoqge7bU8P71hVfz60vD/JXwFTwLeOunJJ0oL1md45DXh4aH22a1uQJOuA44CvLHRfSdLi6BP6maOtFvIiSf4mcBPwT6rq6Tm2b06yO8nuAwcOLOTQkqQF6BP6s8DKofUVwP6+L5DkZcBtwK9W1f+eq09Vbauq6aqanpqa6ntoSdIC9Qn9XcCaJKuTHAdsBHb0OXjX/+PA71fVHxx5mZKkxTBv6FfVIeAq4HbgAeCWqtqTZGuSiwGSnJtkFrgU+GCSPd3uPw+cB1yR5Avdz2uOyTuRJM2rz9k7VNVOYOdI29VDy7sYTPuM7vdh4MNHWaMkaZF4Ra4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQ5b36ZRkPfBbwDLgd6vqfSPbzwN+EzgH2FhVtw5tuxz41W71vVV142IULunZrdpy26RLWDIefN/PTbqERTXvSD/JMuA64CJgLbApydqRbg8BVwAfHdn3R4B3A68H1gHvTvLyoy9bknQk+kzvrANmqmpfVR0EtgMbhjtU1YNVdS/w9Mi+bwb+pKoeqapHgT8B1i9C3ZKkI9An9E8DHh5an+3a+jiafSVJi6xP6GeOtup5/F77JtmcZHeS3QcOHOh5aEnSQvUJ/Vlg5dD6CmB/z+P32reqtlXVdFVNT01N9Ty0JGmh+oT+LmBNktVJjgM2Ajt6Hv924MIkL+++wL2wa5MkTcC8oV9Vh4CrGIT1A8AtVbUnydYkFwMkOTfJLHAp8MEke7p9HwHew+CDYxewtWuTJE1Ar/P0q2onsHOk7eqh5V0Mpm7m2vcG4IajqFGStEi8IleSGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIb1CP8n6JHuTzCTZMsf245Pc3G2/K8mqrv3FSW5Mcl+SB5K8c3HLlyQtxLyhn2QZcB1wEbAW2JRk7Ui3K4FHq+oM4Frgmq79UuD4qjobeB3wzw9/IEiSxq/PSH8dMFNV+6rqILAd2DDSZwNwY7d8K3BBkgAFnJhkOfAS4CDw14tSuSRpwfqE/mnAw0Prs13bnH2q6hDwGHAKgw+A7wBfBx4C/nNVPTL6Akk2J9mdZPeBAwcW/CYkSf30Cf3M0VY9+6wDngJeCawG/nWSH/uBjlXbqmq6qqanpqZ6lCRJOhJ9Qn8WWDm0vgLY/2x9uqmck4BHgMuAT1TV96rqr4A/A6aPtmhJ0pHpE/q7gDVJVic5DtgI7BjpswO4vFu+BLijqorBlM75GTgReAPw5cUpXZK0UPOGfjdHfxVwO/AAcEtV7UmyNcnFXbfrgVOSzADvAA6f1nkd8FLgfgYfHr9XVfcu8nuQJPW0vE+nqtoJ7Bxpu3po+QkGp2eO7vf4XO2SpMnwilxJaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDWkV+gnWZ9kb5KZJFvm2H58kpu77XclWTW07Zwkn0uyJ8l9SX5o8cqXJC3EvKGfZBlwHXARsBbYlGTtSLcrgUer6gzgWuCabt/lwIeBf1FVZwJ/D/jeolUvSVqQPiP9dcBMVe2rqoPAdmDDSJ8NwI3d8q3ABUkCXAjcW1VfBKiqb1bVU4tTuiRpofqE/mnAw0Prs13bnH2q6hDwGHAK8Gqgktye5J4k//boS5YkHanlPfpkjrbq2Wc58JPAucB3gU8lubuqPvWMnZPNwGaA008/vUdJkqQj0WekPwusHFpfAex/tj7dPP5JwCNd+51V9Y2q+i6wE3jt6AtU1baqmq6q6ampqYW/C0lSL31CfxewJsnqJMcBG4EdI312AJd3y5cAd1RVAbcD5yQ5ofsweBPwpcUpXZK0UPNO71TVoSRXMQjwZcANVbUnyVZgd1XtAK4Hbkoyw2CEv7Hb99Ek72fwwVHAzqq67Ri9F0nSPPrM6VNVOxlMzQy3XT20/ARw6bPs+2EGp21KkibMK3IlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGtIr9JOsT7I3yUySLXNsPz7Jzd32u5KsGtl+epLHk/zy4pQtSToS84Z+kmXAdcBFwFpgU5K1I92uBB6tqjOAa4FrRrZfC/zx0ZcrSToafUb664CZqtpXVQeB7cCGkT4bgBu75VuBC5IEIMlbgH3AnsUpWZJ0pPqE/mnAw0Prs13bnH2q6hDwGHBKkhOBXwF+/ehLlSQdrT6hnznaqmefXweurarHn/MFks1JdifZfeDAgR4lSZKOxPIefWaBlUPrK4D9z9JnNsly4CTgEeD1wCVJ/hNwMvB0kieq6reHd66qbcA2gOnp6dEPFEnSIukT+ruANUlWA18DNgKXjfTZAVwOfA64BLijqgr4qcMdkvwa8Pho4EuSxmfe0K+qQ0muAm4HlgE3VNWeJFuB3VW1A7geuCnJDIMR/sZjWbQk6cj0GelTVTuBnSNtVw8tPwFcOs8xfu0I6pMkLSKvyJWkhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkN6hX6S9Un2JplJsmWO7ccnubnbfleSVV37zyS5O8l93b/nL275kqSFmDf0kywDrgMuAtYCm5KsHel2JfBoVZ0BXAtc07V/A/gHVXU2cDlw02IVLklauD4j/XXATFXtq6qDwHZgw0ifDcCN3fKtwAVJUlWfr6r9Xfse4IeSHL8YhUuSFq5P6J8GPDy0Ptu1zdmnqg4BjwGnjPR5K/D5qnryyEqVJB2t5T36ZI62WkifJGcymPK5cM4XSDYDmwFOP/30HiVJko5En5H+LLByaH0FsP/Z+iRZDpwEPNKtrwA+DvxiVX1lrheoqm1VNV1V01NTUwt7B5Kk3vqE/i5gTZLVSY4DNgI7RvrsYPBFLcAlwB1VVUlOBm4D3llVf7ZYRUuSjsy8od/N0V8F3A48ANxSVXuSbE1ycdfteuCUJDPAO4DDp3VeBZwB/PskX+h+fnTR34UkqZc+c/pU1U5g50jb1UPLTwCXzrHfe4H3HmWNkqRF4hW5ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ3pFfpJ1ifZm2QmyZY5th+f5OZu+11JVg1te2fXvjfJmxevdEnSQs0b+kmWAdcBFwFrgU1J1o50uxJ4tKrOAK4Frun2XQtsBM4E1gMf6I4nSZqAPiP9dcBMVe2rqoPAdmDDSJ8NwI3d8q3ABUnStW+vqier6qvATHc8SdIE9An904CHh9Znu7Y5+1TVIeAx4JSe+0qSxmR5jz6Zo6169umzL0k2A5u71ceT7O1Rl/o5FfjGpIuYT66ZdAWakOf97+cL6HfzVX069Qn9WWDl0PoKYP+z9JlNshw4CXik575U1TZgW5+CtTBJdlfV9KTrkObi7+f49Zne2QWsSbI6yXEMvpjdMdJnB3B5t3wJcEdVVde+sTu7ZzWwBvg/i1O6JGmh5h3pV9WhJFcBtwPLgBuqak+SrcDuqtoBXA/clGSGwQh/Y7fvniS3AF8CDgFvr6qnjtF7kSTNI4MBuZaqJJu76TPpecffz/Ez9CWpId6GQZIaYuhLUkMMfUlqSJ/z9PUCleTlwLfKL270PNP9bq5kKIOq6p7JVdQOQ3+JSHI1cEtVfTnJ8cAngB8HDiW5rKr+dLIVSgNJ3gNcAXyF71+hX8D5k6qpJZ69s0Qk2QOcVVXV3dZiE/D3gVcDN1aVN7rT80J3m5Wzuxs4asyc0186Dg5N47yZwd1Nn6qqB/AvOj2/3A+cPOkiWmUYLB1PJjkL+L/ATwO/PLTtxMmUJM3pPwKfT3I/8OThxqq6eHIltcPQXzp+icGzDKaAa7vnF5DkZwG/INPzyY0MHrR0H/D0hGtpjnP6DUjy1qr62KTrkACS3FlVb5p0Ha0y9BuQ5KGqOn3SdUgASd7PYFpnB8+c3vEv0jFweqcNcz3MRpqUn+j+fcNQm6dsjokj/QY40pd0mCP9JSLJfczxKEoGo/xXjLkc6VkleQXwH4BXVtVFSdYCb6yq6ydcWhMc6S8RSeZ6PmYYPKLyXVX1s2MuSZpTkj8Gfg/4d1X1490jVj9fVWdPuLQmeHHWElFVf3n4B3g58HbgM8B7gJ2TrE0acWpV3UJ3umZVHQJ8ot6YOL2zRCR5NYPHVG4CvgnczOAvuZ+eaGHSD/pOklPopiOTvAF4bLIltcPpnSUiydPA/wSurKqZrm1fVf3YZCuTninJa4H/ApzF4JYMU8ClVfXFiRbWCEf6S8dbGYz0P53kE8B2PFVTz097gDcBf5vB7+henGoeG0f6S0ySE4G3MJjmOZ/BJe8fr6pPTrQwqZPknqp67XxtOjYc6S8xVfUd4CPAR5L8CHApsAUw9DVRSf4GcBrwkiQ/wff/En0ZcMLECmuMI31JY5HkcgYPT5kGdg9t+jbwoar6w0nU1RpDX9JYeQPAyTL0JY1Fknc81/aqev+4ammZc/qSxuWHJ12AHOlLUlMc6UsaqySrgX8FrGIog3xc4ngY+pLG7b8B1wP/HR+XOHZO70gaqyR3VdXrJ11Hqwx9SWOV5DJgDYMLBn1c4pg5vSNp3M4G/jGD24Qcnt7xcYlj4khf0lgl+TJwTlUdnHQtLfLOdpLG7YvAyZMuolVO70gat1cAX06yi2fO6XvK5hgY+pLG7d2TLqBlzulLGrskrwLWVNWfJjkBWFZV3550XS1wTl/SWCX5Z8CtwAe7ptMYXLClMTD0JY3b24G/C/w1QFX9BfCjE62oIYa+pHF7cvh0zSTLGZynrzEw9CWN251J3sXgsYk/A/wBg/vwaAz8IlfSWCV5EXAlcCGD5+TeXlW/M9mq2mHoSxqrJL9UVb81X5uODad3JI3b5XO0XTHuIlrlxVmSxiLJJuAyYHWSHUObfhj45mSqao+hL2lc/hfwdeBU4DeG2r8N3DuRihrknL4kNcSRvqSxSPJt5j4fP0BV1cvGXFKTHOlLUkM8e0eSGmLoS1JDDH0tOUn+YZJK8ne69VVJ7l/E4/9ukrXd8ruG2hf1daRjwdDXUrQJ+CywcbEPnGRZVf3TqvpS1/Su59xBep4x9LWkJHkpg9v2XskcoZ/khCS3JLk3yc1J7koy3W3blOS+JPcnuWZon8eTbE1yF/DGJJ9JMp3kfQxuGvaFJB/pui9L8jtJ9iT5ZJKXdMf4TJJrk/yPJA8kOTfJHyb5iyTvPdb/XaTDDH0tNW8BPlFVfw48kuS1I9vfBjxaVecA7wFeB5DklcA1wPnAa4Bzk7yl2+dE4P6qen1VffbwgapqC/D/quo1VfULXfMa4LqqOhP4FvDWodc+WFXnAf8V+CMG95U/C7giySmL9P6l52Toa6nZBGzvlrd368N+8vD2qrqf718Jei7wmao6UFWHgI8A53XbngI+1vP1v1pVX+iW7wZWDW07fOuB+4A9VfX1qnoS2Aes7Hl86ah4cZaWjG60fD5wVpICljG4GOgDw92ebffnOPQTVfVUzzKeHFp+CnjJHNueHun3NP6/qDFxpK+l5BLg96vqVVW1qqpWAl8FVgz1+Szw8wDdGThnd+13AW9KcmqSZQz+Qrizx2t+L8mLF+0dSMeYoa+lZBPw8ZG2j/HMM2w+AEwluRf4FQbTO49V1deBdwKfBr4I3FNVf9TjNbcB9w59kSs9r3kbBjWlG8W/uKqeSPK3gE8Brx5+Zqu0lDmPqNacAHy6m5IJ8C8NfLXEkb4kNcQ5fUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktSQ/w8Y9PUoPaKZwQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "results.groupby('Algorithm').ndcg.mean().plot.bar()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
